Due to the specialized ways that database management systems (DBMS) utilize memory to access data, a DBMS typically implements its own memory management techniques rather than relying on more general memory management techniques that are provided by the underlying operating system on which the DBMS runs. For example, a DBMS may expressly request that the operating system allocate to it a portion of memory so that it can manage such memory on its own, thereby avoiding triggering of memory management techniques (disk swaps, LRU page replacement algorithms, etc.) that the underlying operating system may typically use to “over-commit” its available physical memory in an effort to provide running applications a larger “virtual” memory space in which to execute. That is, although the operating system may provide the DBMS a virtual memory space that is larger than the portion of allocated physical memory requested by the DBMS application, the DBMS application can, using its own memory management techniques, ensure that it utilizes its virtual memory space within the bounds of its allocated physical memory and therefore avoid any operating system level memory management activities that would otherwise adversely affect performance of the DBMS (e.g., untimely writes to swap disk, etc.).
Tuning the DBMS memory has been at the forefront of database research. Recently, autonomic techniques for tuning the memory of DBMS at runtime have been developed for some commercial relational databases. One such technique employs memory pools, each employing specialized paging policies apart from the paging policies of the operating system. Tuning parameters in this technique include the amount of memory to be allocated to these pools and how that memory is to be divided among the various memory pools.
The largest of the memory pools is the buffer pool, which contains the memory pages of database tables that are actively involved in transaction processing. As a transaction modifies rows in database tables, the pages containing these rows are brought into the buffer pool from disk and are modified in place. When the transaction is eventually committed by the DBMS, these “dirty” pages are flushed to disk under the control of the DBMS, for example, by atomically writing a record relating to committed transaction into a write-ahead transaction log on disk to ensure that the transaction's changes are never lost. It should be noted that the DBMS, not the operating system, determines when dirty pages of the buffer pool are written to disk.
In addition, the DBMS, implementing its own memory management, typically maintains its own free list of memory pages and memory page descriptor data structures that are separate and different from any memory management based free memory page lists and data structures maintained by the operating system. Indeed, a memory page that the DBMS may regard as free (e.g., because it has recently completed a database query transaction relating to the data in the memory page) may actually appear to the operating system to be a more important memory page because the DBMS has recently accessed the memory page. As an additional example, memory page descriptor data structures maintained by the DBMS may indicate which memory pages are “clean” and which ones are not. Clean memory pages are those that contain data that matches the corresponding data stored in the database on disk. Because the operating system has no knowledge that the DBMS utilizes portions of its allocated memory as an in-memory cache of the data it stores in the database on disk (e.g., for faster access and query response times), it is not able to similarly characterize the memory pages used by the DBMS.
The DBMS's own memory management techniques referenced above work well when the DBMS is the only application running on a host computer and is able to ensure its own allocation of physical memory. However, when a DBMS is run in a virtual machine that is hosted on a computer with other virtual machines, the host computer's physical memory is managed by virtualization software (sometimes referred to as a hypervisor) that dynamically allocates physical memory among the virtual machines over time, depending upon the particular memory needs of the virtual machines at particular points in time. That is, the hypervisor may over-commit the physical memory of the host computer (sometimes referred to as “machine” memory), providing each virtual machine an illusion that it possesses a certain amount of physical memory (referred to as “guest” physical memory) while allocating actual machine memory to the virtual machine only when it needs it. Over-committing machine memory by the hypervisor in this manner facilitates more efficient use of machine memory because, typically, some virtual machines are lightly loaded while others are more heavily loaded, and relative activity levels vary over time. When the hypervisor experiences memory pressure, for example, because a particular virtual machine requires more machine memory than it is has been allocated, the hypervisor may utilize a variety of techniques to request other virtual machines to “release” allocated machine memory pages back to the hypervisor (for re-allocation to the virtual machine needing more memory). One such technique, known as ballooning, relies on the memory management techniques of the operating systems in the virtual machine (referred to as “guest” operating system) to identify memory pages that may be the best candidates for release to the hypervisor. However, as discussed, if a DBMS application is running in the virtual machine, the operating system's memory management techniques may conflict with the DBMS application's own memory management techniques, thereby resulting in the possible release of memory pages to the hypervisor that may have been important to the DBMS application. Indeed, when the memory needs of the other virtual machines are high, the DBMS, as initially tuned, may be forced to run with insufficient physical or virtual memory available to the virtual machine, resulting in undesirable page thrashing either in the virtual machine or the hypervisor. As a result, the initially tuned parameters for a DBMS may not be applicable to the DBMS running in a virtual machine over the course of time due to the memory over-commitment and management activities of the hypervisor.